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ABSTRACT 

This paper briefly reviews the learning outcome 
aspects of seven recent follow-up studies completed for seven 
different transfer discipline areas at William Rainey Harper College 
(Illinois). It notes an increasing interest by the field of 
institutional research in evaluating how students learn. Faculty in 
the fields of psychology, English, management, speech, geography, the 
honors program, . his tory and political science have been using surveys 
evaluating to what extent students achieve general learning skills. 
Major findings have indicated that Harper College has its greatest 
impact on building self-esteem, developing problem solving skills, 
choosing a career path, communication skills, evaluation of 
information, and library research. The college seems to have little 
effect on building skills to work in small groups. Grades do not seem 
to have a negative effect on learning, and lectures and class 
discussions are perceived to be the two most effective teaching 
delivery systems. Only assigned papers were rated as a clearly 
superior classroom assessment measure. Forces moving faculty to 
become more concerned about evaluating general education learning are 
identified and include national goals on learning, diversity 
pressures, the focus on outcomes of accrediting agencies, and a 
faculty development focus on teaching and learning delivery systems. 
Institutional research is seen to be the obvious partner of faculty 
in developing good surveys and research strategies in learning 
outcomes. Contains one table and seven references. (DB) 
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ABSTRACT 

This paper examines a trend in AIR to begin research in learning outcomes 
in the classroom. The study focuses on the learning outcome bent of 
seven recent follow-up studies completed for seven different transfer discipline 
areas at William Rainey Harper College which is located in Palatine, Illinois. 
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It appears that the field of Institutional Research is now becoming interested for the first 
time in its history in evaluating how students learn. This was demonstrated by Pat Terenzini’s 
monumental study on the key factors affecting critical thinking skills published in Research in 
Higher Education in February of 1995. This was followed by the Forum in Boston in May of 
1995 where the keynote address given by Carol Twigg focused on the need for a national 
learning infrastructure. In addition, four papers focused on evaluating how students learn. When 
the author gave this paper at the Illinois AIR Conference in November of 1995 he asked the 
audience if any were yet involved in learning outcomes research. Not one person said they were 
but some indicated they had talked to faculty who were involved and know that at some point 
they would have to become involved in this type of research. 

Faculty in transfer oriented classes at Wm. Rainey Harper College are now, for the first 
time, addressing questions on how students learn in their classroom for their Program Review. 
Studies for Psychology, English, Honors, Management, Speech, Geography, History and 
Political Science all focus on to what extent students achieve general learning skills. Many 
questions focused on what most contributed to students’ learning at Harper and areas needing 
improvement. Other questions focused on general cognitive skills such as various 
communication skills, problem solving, critical thinking, ability to work in groups, etc. Other 
items focused on the development of reading habits, self-esteem, leadership friendships, 
networks, global concerns and volunteering in the community. Some items asked about how 
grading affected the learning process. Still other items evaluated various teaching delivery 
systems and various classroom assessment techniques. Another concern raised was the value 
system of students in determining what success in college meant. 
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The major findings indicated Harper College had its greatest impact on building self- 
esteem, developing problem solving skills gaining help in choosing a career path, 
communication skills, evaluation of information and library research. Harper College seemed to 
have the least effect on building skills to work in small groups even though this was thought to 
be important. Grades do not seem to have a negative effect on learning with the exception of 
experimenting with different learning styles. Lectures are one of the two most perceived 
effective teaching delivery systems along with class discussion. Multiple choice testing as 
classroom assessment was rated effective, very close to essay and short answer testing. Only 
assigned papers were rated as a clearly superior classroom assessment. Honors students do not 
seem to gain much more in learning than ordinary students with the exception of the types of 
transfer colleges chosen. 

In the last two years a number of items in follow-up studies have been focusing on 
learning outcomes in transfer related disciplines at Harper College. Some of the types of items 
by discipline are as follows: 

Table I 

Types of Learning Outcome Follow-up Survey Items by Discipline 
Discipline Follow-up Survey Item Type 

1- Transfer Management - Thirty-one management skills listed - asks both how 

important are these skills for their job and how well 
prepared were they in this skill area. 
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Table I (continued) 

Types of Learning Outcome Follow-up Survey Items by Discipline 
Discipline Follow-up Survey Item Type 

2- Speech - Eight speaking and theatre skills - ask how frequently 

they use them in their current situation and how well 
prepared were they in this skill area. 

3- Geography /History /Political - Class size experienced and evaluation of how class size 

Science 

affected learning process. 

- Twenty descriptions of learning environment rating of 
each environment at Harper and at the college to which 
they transferred. 

- Five teaching methods and eight assessment listed asked 
how frequently they encountered these and how effective 
they were. 

4- Psychology - Nineteen possible ways psychology may have influenced 

their life plan. 

- Experiences with volunteer projects 

- Class size and effect on learning 

- Ten teaching methods eight means of assessment 
How frequently encountered - how effective each was 




Table I (continued) 

Types of Learning Outcome Follow-up Survey Items by Discipline 



Discipline 

5- Honors Program 



6- English! 



7- English II 



Follow-up Survey Item Type 

- Types of College transferred to 

- Affect on leadership style, creativity, self-esteem, friend- 
ship networks developed, awareness of global concerns, 
reading habits, oral reporting, writing skills, teamwork, 
succeeding in life, level of motivation, competitiveness, 
desire to leam further. 

- Factors that most contributed to their learning 

- How writing topics were chosen and why 

- How grades affect how students write 

- Comfort in working in group writing projects 

- Perceived importance of grades and its fairness 

- Preference among three ways of teaching English 
composition 

- Contributions to success in college 

- Perception of what contributes to high grades in English 

- Eight English composition skill areas - frequency of use 
and degree of preparation 

- Perception of fairness of level of English placed in by 
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In each of these follow-up surveys faculty discussed the results of these studies to decide 
how they could modify their instructional delivery to improve the learning outcomes. 

There are a number of forces at work which are moving faculty to become more 
concerned about evaluating the general education learning taking place in the classroom: 

a) National goals on learning 

b) Diversity pressures 

c) Accrediting agencies focus on outcomes 

d) Emphasis on distance learning 

e) Explosion of technology to support learning environments 

f) Pressures created by unprepared students 

g) Faculty development change of focus to teaching and learning delivery 
systems 

h) Movement from hierarchical organizations to TQM team approaches 

i) Breakthrough developments of new learning environments. 

These break through wholistic learning environments involve either groups of courses the 
same students are all enrolled in or simulated organizational learning environments. All these 
environments have the following in common: The instructor no longer the sole authority, 
students having more responsibility for their learning, cross discipline learning, intense group 
work, and high student enthusiasm. 

As all these forces come to a head at the same time faculty look around for help in 
evaluating and measuring these outcomes. Institutional research offices are a natural place to 
look even though we traditionally have not done this type of research. There are a number of 
challenges in doing this type of research. First the possibilities for criteria as measured by 
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behavior and attitude changes are very large. It will take some time to develop a complete list of 
criteria survey items. Just as challenging is moving from evaluative and attitudinal items to 
behavioral based items which will lend more credibility to the results. For example, instead of 
asking respondents how they feel about reading or how competent they feel in reading, questions 
would center on how often they read, the type of material they read and how they use their 
reading. Another critical issue is determining if the behavioral or attitude changes are permanent 
or simply reflect the emotions of the class. This can only be determined by surveying students 
several years after the class or event. 

Besides measuring learning changes by survey it will be important to explore testing 
samples of students to independently determine if certain learning changes have taken place. 

This research could also examine if there are strong relationships between these test results and 
survey results. Testing students after the fact is expensive and difficult to motivate students to 
objectively take tests which are not required by their program. 

Even more challenging is to develop comparison groups which would allow more 
meaningful interpretation to changes in behavior, attitudes, evaluations or in test results. Many 
times there is no true comparison group available so a less meaningful quasi comparison group 
must be considered. In addition, comparison groups are difficult to construct to provide 
comparable experiences and ability level so that only the teaching strategy or intervention in 
question varies. Faculty are often reluctant to allow the use of comparison groups as it can seem 
very threatening. 

Finally, this type of research requires the examination of the magnitude of die surve> 01 
test differences in relationship to the additional resources required by a new delivery system or 
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intervention. This implies a panel of persons representing both the academic and resource sides 
of the institution making judgments concerning the costs and benefits of any given strategy. 
Thus, let the dialogue begin on research strategies so we, as institutional research professionals, 
can prepare for the learning outcome studies immediately ahead of us. 
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